Vert: an Automatic Summary Evaluation System

In this paper, we propose an automatic evaluation procedure based on a metric which could provide summary evaluation without human assistance. Our system, which includes two metrics, for summary evaluation, is presented. The first metric is based on a known and powerful statistical test, the χ goodness-of-fit test, and has been used in several applications. The second metric is derived from three common metrics used to evaluate NLP systems, namely precision, recall and f-measure. The combination of these two metrics is intended to allow one to assess the quality of summaries quickly, cheaply and without the need of human intervention, minimizing though, the role of subjective judgment and bias.